Provable Dictionary Learning via Column Signatures

In dictionary learning, also known as sparse coding, we are given samples of the form y = Ax where x ∈ R is an unknown random sparse vector and A is an unknown dictionary matrix in Rn×m (usually m > n, which is the overcomplete case). The goal is to learn A and x. This problem has been studied in neuroscience, machine learning, vision, and image processing. In practice it is solved by heuristic algorithms and provable algorithms seemed hard to find. Recently, provable algorithms were found that work if the unknown feature vector x is √ nsparse or even sparser. [SWW12] did this for dictionaries where m = n; [AGM13] gave an algorithm for overcomplete (m > n) and incoherent matrices A; and [AAN13] handled a similar case but with somewhat weaker guarantees. This raised the problem of designing provable algorithms that allow sparsity √ n in the hidden vector x. The current paper designs algorithms that allow sparsity up to n/poly(log n). It works for a class of matrices where features are individually recoverable, a new notion identified in this paper that may motivate further work. The algorithm runs in quasipolynomial time because it uses limited enumeration. ∗Princeton University, Computer Science Department and Center for Computational Intractability. Email: [email protected]. This work is supported by the NSF grants CCF-0832797, CCF-1117309, CCF-1302518, DMS-1317308, and Simons Investigator Grant. †Google Research NYC. Email: [email protected]. Part of this work was done while the author was a Postdoc at EPFL, Switzerland. ‡Microsoft Research. Email: [email protected]. Part of this work was done while the author was a graduate student at Princeton University and was supported in part by NSF grants CCF-0832797, CCF-1117309, CCF-1302518, DMS-1317308, and Simons Investigator Grant. §Princeton University, Computer Science Department and Center for Computational Intractability. Email: [email protected]. This work is supported by the NSF grants CCF-0832797, CCF-1117309, CCF-1302518, DMS-1317308, and Simons Investigator Grant.